A Novel Brain-Based Approach for Multi-Modal Multi- Target Tracking in a Mixed Reality Space

We propose an architecture for multi-modal multi-target tracking, for the integration of multi-sensory input and its top-down modulation through active deployment of sensors and effectors based on Bayesian inference. The implementation of our architecture, which is inspired by the Superior Colliculus (SC), engages a method for dynamical allocation of sensors and effectors to enhance tracking and to resolve conflicting multi-modal data. We suggest to use the joint probabilistic data association method as the basis for facilitating the topdown modulation of bottom-up sensory data. A world model, which is automatically created, provides high level knowledge of the current state and interaction scenarios, which is then used for modulation of bottom-up sensory data. We test our SC-based framework for multi-sensory data fusion to tackle in real-time a multi-person tracking problem in a human accessible mixed reality environment called XIM (eXperience Induction Machine).